-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Flag and maybe delete messages after messages have been copied #9546
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
87b78c6
to
81ee4c2
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please, consider to run gradlew spring-integration-mail:check
locally.
Otherwise we have those Checkstyle failures:
Error: eckstyle] [ERROR] /home/runner/work/spring-integration/spring-integration/spring-integration-mail/src/main/java/org/springframework/integration/mail/AbstractMailReceiver.java:516:17: '}' at column 3 should be alone on a line. [RightCurly]
> Task :spring-integration-mail:checkstyleMain
> Task :spring-integration-mail:checkstyleMain FAILED
> Task :spring-integration-stream:compileTestJava
Error: eckstyle] [ERROR] /home/runner/work/spring-integration/spring-integration/spring-integration-mail/src/test/java/org/springframework/integration/mail/ImapMailReceiverTests.java:1053:17: Redundant 'public' modifier. [RedundantModifier]
Also, ad your name to all the affected classes.
We will consider to back-port it, but that is not going to happen for 6.1.x
.
That one is out of Open Source support already.
setMessageFlagsAndMaybeDeleteMessages(originalMessages); | ||
} | ||
else { | ||
setMessageFlagsAndMaybeDeleteMessages(filteredMessages); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
May we consider a local method variable for Message[]
instead of else
branch with the same method call?
I mean something like:
Message[] messages = filteredMessages;
if (this.headerMapper == null && (this.autoCloseFolder || this.simpleContent)) {
messages = new Message[filteredMessages.length];
...
}
setMessageFlagsAndMaybeDeleteMessages(messages);
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was wanted to do that initially as well. However, in the if
we are creating a new Message
of the IntegrationMimeMessage
. This means that when we call setMessageFlagsAndMaybeDeleteMessages
and we invoke Message#setFlags
the actual implementation differs. If we look into the IMAPMessage#setFlags
the implementation looks like
synchronized (getMessageCacheLock()) {
try {
IMAPProtocol p = getProtocol();
checkExpunged(); // Insure that this message is not expunged
p.storeFlags(getSequenceNumber(), flag, set);
} catch (ConnectionException cex) {
throw new FolderClosedException(folder, cex.getMessage());
} catch (ProtocolException pex) {
throw new MessagingException(pex.getMessage(), pex);
}
}
whereas the MimeMessage#setFlags
looks like
if (set)
flags.add(flag);
else
flags.remove(flag);
I am going to add a comment there to explain why it is done in the way I did it
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I’m not sure why does that matter since you call setMessageFlagsAndMaybeDeleteMessages
in both cases? So, one local variable for both cases feels right.
Ok, I’ll look into that locally on merge.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@artembilan it matters a lot because what will be invoked in the end is not the same. The behaviour of the setFlags
will change due to the fact that there will be different implementations.
Prior to this PR the setFlags
was invoked on the original messages received from the folder. The goal of the PR is to still invoke the setFlags
on the original messages. However, instead of doing it before copying it, to do it after it has been copied.
I have an idea for a test case. I'll add it to illustrate the problem.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am still failing to understand why this code is not OK with you:
private void postProcessFilteredMessages(Message[] filteredMessages) throws MessagingException {
Message[] messagesToProcess = filteredMessages;
// Copy messages to cause an eager fetch
if (this.headerMapper == null && (this.autoCloseFolder || this.simpleContent)) {
messagesToProcess = new Message[filteredMessages.length];
for (int i = 0; i < filteredMessages.length; i++) {
Message originalMessage = filteredMessages[i];
messagesToProcess[i] = originalMessage;
MimeMessage mimeMessage = new IntegrationMimeMessage((MimeMessage) originalMessage);
filteredMessages[i] = mimeMessage;
}
}
setMessageFlagsAndMaybeDeleteMessages(messagesToProcess);
}
You see, we still fulfill that messagesToProcess
with original messages.
And then call setMessageFlagsAndMaybeDeleteMessages()
only against a single source of truth.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry @artembilan, I misunderstood you. I know what you mean now. The messages
will be the original messages populated in the loop.
I adjusted the PR. Please correct me if I am wrong.
Thanks for the quick review @artembilan.
That was my mistake. The checkstyle errors are already fixed.
Done
We are already upgraded to Spring Boot 3.3 and using Spring Integration 6.3, so back-porting it to at least 6.3 would be more than appreciated for us. |
|
||
@Override | ||
public void writeTo(OutputStream os) throws IOException, MessagingException { | ||
if (this.exceptionsBeforeWrite.decrementAndGet() >= 0) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like just AtomicBoolean
will be enough without any extra ctor arg.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You are right. I've adjusted it
setMessageFlagsAndMaybeDeleteMessages(originalMessages); | ||
} | ||
else { | ||
setMessageFlagsAndMaybeDeleteMessages(filteredMessages); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am still failing to understand why this code is not OK with you:
private void postProcessFilteredMessages(Message[] filteredMessages) throws MessagingException {
Message[] messagesToProcess = filteredMessages;
// Copy messages to cause an eager fetch
if (this.headerMapper == null && (this.autoCloseFolder || this.simpleContent)) {
messagesToProcess = new Message[filteredMessages.length];
for (int i = 0; i < filteredMessages.length; i++) {
Message originalMessage = filteredMessages[i];
messagesToProcess[i] = originalMessage;
MimeMessage mimeMessage = new IntegrationMimeMessage((MimeMessage) originalMessage);
filteredMessages[i] = mimeMessage;
}
}
setMessageFlagsAndMaybeDeleteMessages(messagesToProcess);
}
You see, we still fulfill that messagesToProcess
with original messages.
And then call setMessageFlagsAndMaybeDeleteMessages()
only against a single source of truth.
@@ -1063,6 +1067,21 @@ public void writeTo(OutputStream os) throws IOException, MessagingException { | |||
} | |||
super.writeTo(os); | |||
} | |||
|
|||
@Override | |||
public synchronized void setFlags(Flags flag, boolean set) throws MessagingException { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do we need to override this if there is that Message.getFlags()
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is to show that we are calling the setFlags
on the original message and not the copied one. The Flags
can be shared. The MimeMessage
has
flags = source.getFlags();
if (flags == null) // make sure flags is always set
flags = new Flags();
So if the source has Flags
and we call the setFlags
on the filteredMessages
then the test will still pass. The goal of the change is to show that the setFlags
is called on the original message.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Doesn't look like you validate somehow in the test if message was original or IntegrationMimeMessage
.
Therefore this extra precaution is redundant.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is
assertThat(msg2.getMyTestFlags().contains(Flag.SEEN)).isTrue();
In the test.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But looks like it is exactly the same as previous one:
assertThat(msg2.getFlags().contains(Flag.SEEN)).isTrue();
That's why I believe that is really enough.
Because you are not going to deal with mock messages in the real application.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added this to validate that the setFlags
is called on the original message and not on the IntegrationMimeMessage
. Currently, GreenMailUtil#newMimeMessage
does not set the flags. However, if you want I can set it up in such case that if the setMessageFlagsAndMaybeDeleteMessages
is called on the IntegrationMimeMessage
the test would fail.
assertThat(msg2.getFlags().contains(Flag.SEEN)).isTrue();
The test above passes as the Flags
are shared. But
assertThat(msg2.getMyTestFlags().contains(Flag.SEEN)).isTrue();
would fail as it has it's own flags. If you think it is too much, I can remove it
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK. You probably can spy()
on the GreenMailUtil#newMimeMessage
result to verfiy that exactly its setFlags()
is called.
It is still look redundant to have our own property when it is there in the super class.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
good point with the spy()
. I'll add that
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've replaced that with a spy. Hope it is good now.
@@ -1063,6 +1067,21 @@ public void writeTo(OutputStream os) throws IOException, MessagingException { | |||
} | |||
super.writeTo(os); | |||
} | |||
|
|||
@Override | |||
public synchronized void setFlags(Flags flag, boolean set) throws MessagingException { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Doesn't look like you validate somehow in the test if message was original or IntegrationMimeMessage
.
Therefore this extra precaution is redundant.
@@ -1063,6 +1067,21 @@ public void writeTo(OutputStream os) throws IOException, MessagingException { | |||
} | |||
super.writeTo(os); | |||
} | |||
|
|||
@Override | |||
public synchronized void setFlags(Flags flag, boolean set) throws MessagingException { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But looks like it is exactly the same as previous one:
assertThat(msg2.getFlags().contains(Flag.SEEN)).isTrue();
That's why I believe that is really enough.
Because you are not going to deal with mock messages in the real application.
@filiphr , thank you for contribution; looking forward for more! |
Fixes: #9546 Issue link: #9546 Sometimes the IMAP connection breaks just after the message flags have been set, but the message has not been copied yet. This then leads to the message never being received by (as it has been flagged and the next search will not return it). * Flag and maybe delete messages after messages have been copied **Auto-cherry-pick to `6.2.x`** # Conflicts: # spring-integration-mail/src/main/java/org/springframework/integration/mail/AbstractMailReceiver.java
Fixes: #9546 Issue link: #9546 Sometimes the IMAP connection breaks just after the message flags have been set, but the message has not been copied yet. This then leads to the message never being received by (as it has been flagged and the next search will not return it). * Flag and maybe delete messages after messages have been copied # Conflicts: # spring-integration-mail/src/main/java/org/springframework/integration/mail/AbstractMailReceiver.java # Conflicts: # spring-integration-mail/src/test/java/org/springframework/integration/mail/ImapMailReceiverTests.java
Thanks a lot for the speedy review and integration @artembilan |
The reason for this pull request is due to the fact that we have noticed that sometimes the IMAP connection breaks just after the message flags have been set, but the message has not been copied yet. This then leads to the message never being received by (as it has been flagged and the next search will not return it).
e.g.
I could reproduce a similar exception while debuging in the
IntegrationMimeMessage
constructor. If I put a break point there, wait a minute and then continue I would get this exception.While debugging I could find some things such as
I know that it says Spring Integration 6.1.8 here. However, I could reproduce the same with 6.3.5 as well.
I've added a test trying to mimic the behaviour (an
IOException
) when copying the message contents.